Posted on 2022-05-11 | Last updated 2022-05-11 | Tags: ProgrammingWeb-Development

These last weeks I have been working on a benchmark of different REST frameworks, with the constraint that they are capable of generating OpenAPI documents by annotating code. The github repo can be found here

Some motivation first; At work we have an Apache Solr backend, containing documents that are shown in our front-end applications. These documents contain sensitive registry data, and therefore need to be anonymized before reaching the front-end client. We therefore need to have a thin layer in front of our Solr server, that makes sure only data that we actually are allowed to show out is shown.
I wanted to see which alternatives were out there, and which one was the best to achieve a balance between performance, and ease of writing code.

---

Requirements

First, I had to devise a list of requirements that the frameworks were supposed to be able to perform, and secondly decide what types of benchmarks I would be running. I ended up with the following requirements

• Generate an OpenAPI document by annotating code
  
  • Summary for a route
  • Query parameter description
  • Example of the response
  • Multiple responses
• Validation of inputs, (aka query and path parameters)
• Serialization of JSON documents, supporting what's known as "ignore null keys"

---

Benchmarks

Given the preceding requirements I came up with three benchmarks

Hello World

Should return a text/plain, 200 OK response, with a UTF-8 encoded string saying "Hello World".
This benchmark is meant to test how the framework responds under ideal circumstances, without any overhead.

JSON Serialization

Takes a query parameter document_type, which is an int. This should have an example, and be required in the Swagger-UI. If a client sends something other than an int, a 4XX error should be returned.
After validating the parameter, the framework should send a request to the Solr backend, requesting the relevant documents, deserializing them, and reserializing them before returning the JSON response to the client.
This benchmark is meant to show how fast the framework is at JSON deserialization and serialization.

Anonymization

Like the previous benchmark, but before returning the response to the client, should iterate through all the documents, and anonymize the data.
This benchmark is meant to show the overhead of the framework if it needs to do additional processing on the documents.

---

Execution

Our application stack, and the industry as a whole is moving to containerization and Kubernetes. It therefore felt prudent to use docker containers as the execution environment of the frameworks. It also gave the additional benefit of minimal setup for running the benchmarks. Anyone can clone the repo and run

docker-compose up --build --force-recreate

to execute the benchmarks.
The second thing to think about was resource limitation. I ran the benchmark on a 16 core Ryzen Threadripper, with 128 GB of ram. This is not a representative environment for container-orchestration, where you usually want multiple containers with a more limited resource pool per instance.
Luckily docker-compose supports resource limitation^[1], so I ended up limiting the instances to 4 cores, and 4 GB of RAM, simulating a real-world scenario better.

Results

Hello World

Framework	Language	Requests per second	Percent
Actix	Rust	215827	100.0
Asp.Net Core	C#	86755	40.2
Oat++	C++	86166	39.9
Jooby	Java	48200	22.3
NestJS-Fastify	Typescript	17943	8.3
FastAPI	Python	7548	3.5
Flask-Restx	Python	2248	1.0
API Platform Nginx-FPM	PHP	1169	0.5
API Platform Apache	PHP	1094	0.5

JSON Serialization

Framework	Language	Requests per second	Percent
Actix	Rust	2288	100.0
Asp.Net Core	C#	1007	44.0
Jooby	Java	948	41.4
Oat++	C++	432	18.9
NestJS-Fastify	Typescript	178	7.8
Flask-Restx	Python	75	3.3
FastAPI	Python	51	2.2
API Platform Nginx-FPM	PHP	33	1.4
API Platform Apache	PHP	31	1.4

Anonymization

Framework	Language	Requests per second	Percent
Actix	Rust	2219	100.0
Jooby	Java	1060	47.8
Asp.Net Core	C#	945	42.6
Oat++	C++	431	19.4
NestJS-Fastify	Typescript	179	8.1
Flask-Restx	Python	74	3.3
FastAPI	Python	53	2.4
API Platform Nginx-FPM	PHP	33	1.5
API Platform Apache	PHP	31	1.4

---

Analysis

To no one's surprise there is a pretty clear correlation where low-level languages are fast, high-level languages are slow. Rust's Actix crushes the competition in all categories, while interpreted languages like PHP and Python are the slowest. Languages running bytecode on a virtual machine like Java and C# operate somewhere in the middle. I will now go through and look at how different frameworks solved having to create OpenAPI documentation.

Asp.Net Core

/// <summary>Serializing  a json document</summary>
/// <param name="document_type">
/// Some example values: <ul><li><code>1</code></li></ul>
/// </param>
[HttpGet]
[Route("json_serialization")]
[ProducesResponseType(StatusCodes.Status200OK, Type = typeof(List<Entity>))]
[ProducesResponseType(StatusCodes.Status400BadRequest)]
public async Task<IActionResult> JSONSerialization([FromQuery, BindRequired] int document_type) {
    ...
}

Asp.Net Core takes advantage of the built in XML documentation format provided by C#, in addition to attributes to create OpenAPI documentation. The method in itself uses a normal function taking normal parameters, with additional attributes to describe to OpenAPI how to document.

public class Entity {
    public string id { get; set; }
    public int document_type { get; set; }
    public string[] string_array { get; set; }
    public int[] int_array { get; set; }
    public SubEntity[] child_objects { get; set; }
}

The entity being made into a response is described as a normal C# class, with relations embedded into the class as type information.

Asp.Net Core is a solid framework. It is quite popular at the moment, regarded as one of the monoliths. Microsoft is behind it, which can be a blessing or a curse depending on who is leading them at the moment. Right now with Ballmer out, and Microsoft seeming to be driving open-source rather than hindering it, having a big organization behind your framework, with professional developers working full time maintaining it, should keep it going for the forseeable future.

FastAPI

@app.get("/json_serialization", response_model=List[Entity], response_model_exclude_none=True,
         summary='Serializing  a json document')
async def json_serialization(document_type: int) -> List[Entity]:
    """
    :param document_type: Some example values: <ul><li><code>1</code></li></ul>
    """
    ...

Like Asp.Net Core, FastAPI tries to use the built in features of the language rather than inventing new ones. You have attributes over a standard python function describing the endpoint, with pydoc annotations for the parameters.

class Entity(BaseModel):
    id: str
    document_type: int
    string_array: List[str]
    int_array: List[int]
    child_objects: Optional[List[SubEntity]] = None

Entities are described using the type hints added in python 3.5. You therefore get good syntax for describing the entities.

If you want a framework that is easy to use, but don't really care too much for performance, FastAPI is a good alternative. It seems to be on an upswing at the moment.

Actix

#[derive(Serialize, Deserialize, Apiv2Schema)]
struct Info {
    /// Some example values: <ul><li><code>1</code></li></ul>
    document_type: i32,
}
#[api_v2_errors(code=400, description = "Bad request")]
pub struct BadRequest {}
    ...

#[api_v2_operation(summary = "Serializing a json document")]
async fn json_serialization(info: web::Query<Info>, client: web::Data<Client>) -> Result<Json<Vec<Entity>>, BadRequest> {
    ...
}

Actix with the paperclip plugin is a bit more verbose than the previous frameworks. Like them it uses attributes and a controller function. However unlike them you cannot use the function parameters directly.

#[derive(Serialize, Deserialize, Apiv2Schema)]
struct Entity {
    id: String,
    #[serde(skip_serializing_if = "Option::is_none")]
    document_type: Option<i32>,
    #[serde(skip_serializing_if = "Option::is_none")]
    string_array: Option<Vec<String>>,
    #[serde(skip_serializing_if = "Option::is_none")]
    int_array: Option<Vec<i32>>,
    #[serde(skip_serializing_if = "Option::is_none")]
    child_objects: Option<Vec<SubEntity>>
}

Actix using the Serde JSON serialization library is the fastest framework out there. I have tried using it in another project, but found Rust very hard to simply pick up. If you need to squeeze every bit of performance out of your application, Actix could be a good option.

Flask-Restx

json_serialization_parser = reqparse.RequestParser()
json_serialization_parser.add_argument("document_type", type=int,
                                       required=True)

@api.route('/json_serialization')
class JSONSerialization(Resource):
    @api.expect(json_serialization_parser)
    @api.response(400, 'Validation Error')
    @api.marshal_with(entity, skip_none=True)
    def get(self):
        """Serializing  a json document"""
        args = json_serialization_parser.parse_args()
        ...

entity = api.model('Entity', {
    'id': fields.String,
    'document_type': fields.Integer,
    'string_array': fields.List(fields.String),
    'int_array': fields.List(fields.Integer),
    'child_objects': fields.List(fields.Nested(sub_entity))
})

Flask is a well known, established framework. It seems that Flask-Restx is the newest attempt at OpenAPI support, although it has not had a commit in a year at this point.
In general, flask is an easy to write, simple framework. The Restx part seems to me however to be inferior to FastAPI. Instead of native classes, and type hints to generate OpenAPI, it instead opts to creating models as custom objects, making them hard to use if somewhere else in your application you also need the same model. It also opts for having parameters outside the actual function, in a request parser.
In general, due to FastAPI's existence and comparable performance I don't really see a reason to use Flask with Restx.

API Platform

/**
 * Class Entity
 *
 * @package App\Entity
 *
 *
 * @ApiResource(
 *      collectionOperations={
 *         "json_serialization"={
 *              "method" = "GET",
 *              "path"="/json_serialization",
 *              "filters"={
 *                  App\Filter\DocumentType::Class,
 *              },
 *              "openapi_context" = {
 *                  "summary" = "Serializing  a json document",
 *                  "tags" = {"Default"},
 *                  "responses" = {
 *                      "400" = {
 *                          "description" = "Invalid input"
 *                      }
 *                 }
 *              }
 *
 *         }
 *     },
 *     itemOperations={
 *         "get"={
 *             "path"="/entity/{id}",
 *             "openapi_context" = {
 *                  "tags" = {"Default"},
 *                  "summary" = "Not implemented"
 *             }
 *         },
 *
 *     },
 *     normalizationContext={
 *         "skip_null_values" = true
 *     },
 *     attributes={
 *          "pagination_enabled"=false
 *     },
 *     formats={"json"}
 * )
 */


class Entity
{
    /**
     * @var string $id Identifier for result
     *
     * @ApiProperty(identifier=true)
     */
    public $id;
    /** @var int $document_type */
    public $document_type;

    /** @var string[] $string_array */
    public $string_array;

    /** @var int[] $int_array */
    public $int_array;

    /** @var SubEntity[] $child_objects */
    public $child_objects;
}

The weirdest of the bunch is definitely API Platform; A PHP framework. Instead of having controller methods it uses entities as the base for routes, differentiating in itemOperations and collectionOperations. ItemOperations have to exist, but cannot have more parameters than an ID path, making it useless for this benchmark. So you have to use CollectionOperations. A CollectionOperation must return an array of something, so you are stuck forced to use an ItemOperation with only one parameter, or forced to return an array with a CollectionOperation.
Maybe they have a good reason for designing it like this, but out of all the frameworks I used API-Platform was the only one with such a restrictive design. Using comment annotations for describing routes and the OpenAPI structure is also a less than ideal solution, but probably the best you can have in PHP^[2]

class DocumentType implements FilterInterface
{
    public function getDescription(string $resourceClass): array {
        return [
            "document_type" => [
                'property' => NULL,
                'type' => 'int',
                'is_collection' => FALSE,
                'required' => TRUE,
                'description' => "Some example values: <ul><li><code>1</code></li></ul>"
            ],
        ];
    }
}

Path and Query parameters have to be described in their own classes, and imported into the entity using annotations, also a peculiar design choice.

class EntityDataProvider  implements ContextAwareCollectionDataProviderInterface, RestrictedDataProviderInterface, ItemDataProviderInterface, SerializerAwareDataProviderInterface
{
    use SerializerAwareDataProviderTrait;
    public function getCollection(string $resourceClass, string $operationName = NULL, array $context = []) {
        $filters_raw = $context['filters'] ?? [];
        $filters_raw = array_change_key_case($filters_raw, CASE_LOWER);
        switch ($operationName) {
            case 'json_serialization':
                $repo = new JSONSerializationRepository($this->getSerializer());
                break;
            case 'hello_world':
                die('Hello World');
        }
        return $repo->getAll($filters_raw);
    }

    public function getItem(string $resourceClass, $id, string $operationName = NULL, array $context = []) {
        throw new NotFoundHttpException();
    }

    public function supports(string $resourceClass, string $operationName = NULL, array $context = []): bool
    {
        return $resourceClass === Entity::class || $resourceClass == HelloWorldEntity::class;
    }
}

The code that actually executes is in what's called a DataProvider. This function gives you the route name, and the parameters, and it's up to you to deal with it. If you look closely, you will see

die('Hello World')

which was the only way to return a plain/text response. If you decide to go with the API Platform route, be ready to fight the framework at every step of the way, and when that does not work, break it and use standard PHP to do what you want.

Jooby

    @GET(value = "json_serialization")
    @Operation(
            summary = "Serializing  a json document"
    )
    @Parameters({
            @Parameter(description = "Some example values: <ul><li><code>1</code></li></ul>", required = true, name = "document_type")
    })
    @ApiResponse(responseCode = "400")
    public Entity[] json_serialization(@QueryParam int document_type) throws JsonProcessingException {
        ...
    }

public class Entity {
    public String id;
    public int document_type;
    public String[] string_array;
    public int[] int_array;
    public SubEntity[] child_objects;
}

Jooby is a framework written in Java. It seems to have a good interface, in line with FastAPI and Asp.Net Core. It is also fast to boot.

Oat++

    #include OATPP_CODEGEN_BEGIN(ApiController)
    ENDPOINT_INFO(json_serialization) {
        info->summary = "Serializing a json document";
        info->addResponse<Object<Entity>>(Status::CODE_200, "application/json");
        info->addResponse<String>(Status::CODE_400, "text/plain");
        info->queryParams.add<Int32>("document_type").description = "Some example values: <ul><li><code>1</code></li></ul>";
    }
    ENDPOINT_ASYNC("GET", "/json_serialization", json_serialization) {
    ENDPOINT_ASYNC_INIT(json_serialization)

        Action act() override {
            ...
        }

#include OATPP_CODEGEN_BEGIN(DTO)
class Entity : public oatpp::DTO {

    DTO_INIT(Entity, DTO)

    DTO_FIELD(String, id);
    DTO_FIELD(Int32, document_type);
    DTO_FIELD(List<String>, string_array);
    DTO_FIELD(List<Int32>, int_array);
    DTO_FIELD(List<Object<SubEntity>>, child_objects);

};
#include OATPP_CODEGEN_END(DTO)

Oat++ is a C++ framework. It has quite an amount of setup code to get working, and is not very comfortable to write. This is however a failure of C++, and not the framework. C++ does not have any form for built in reflection or attributes, you therefore have to rely on macros to do that kind of work. From a C++ developer's perspective I really like how it is done within the constraints of the language, but if you are not a C++ developer I would stay far away.

NestJS-Fastify

class GetTypeQuery {
  @ApiProperty({
    description: `Some example values: <ul><li><code>1</code></li></ul>`,
  })
  @IsNumberString()
  document_type: number;
}

@Controller()
export class AppController {
  constructor(private readonly appService: AppService) {}

  @Get('json_serialization')
  @ApiOperation({ summary: 'Serializing  a json document' })
  @ApiOkResponse({
    type: Entity,
    isArray: true,
    description: 'Serializing  a json document',
  })
  @ApiBadRequestResponse({ description: 'Bad Request' })
  async json_serialization(@Query() params: GetTypeQuery): Promise<Entity[]> {
    ...
  }

}

export class Entity {
  @ApiProperty()
  id: string;
  @ApiProperty()
  type: number;
  @ApiProperty({ type: [String] })
  string_array: string[];
  @ApiProperty({ type: [Number] })
  int_array: number[];
  @ApiProperty({ type: [SubEntity] })
  child_objects: SubEntity[];
}

NestJS is a typescript framework. My first inpression was that although it supports attributes, it seems to be a bit overzealous with using them. In general though, it seems okay to write with, giving a performance edge over the other interpreted languages.

Conclusion

From a performance perspective, Actix, Asp.Net Core and jooby seems to give you the most bang for the buck. From a usability perspective, my favorites are FastAPI and Asp.Net Core.
Therefore, given Asp.Net Core for more a serious production-grade framework, and FastAPI for smaller, or less performance-dependent applications I would say that you have the frameworks that you are looking for. They are both very easy to write, and seem to be very popular.

Rust seems to be getting more and more popular, but it might be too low-level to be worth learning just for an API. If you are already proficient, or need to squeeze every bit of performance out of your API however, Actix is the fastest one out there.

If you haven't already, the TechEmpower benchmarks give an alternative view of the specified frameworks. And if you see an obvious candidate that supports the requirements given, or have distaste for one of my implementations (and it's not years into the future), please reach out and I'll see if i can add/fix it.